Apparatus and method for searching for building based on image and method of constructing building search database for image-based building search

ABSTRACT

An apparatus and method for searching for a building on the basis of an image and a method of constructing a building search database (DB) for image-based building search. The method includes constructing a building search DB, receiving a query image from a user terminal, detecting a region to which a building belongs in the query image, extracting features of the region detected in the query image, and searching the building search DB for a building matching the extracted features. Therefore, building search performance can be improved.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application No.10-2018-0048387, filed on Apr. 26, 2018 in the Korean IntellectualProperty Office (KIPO), the entire content of which is herebyincorporated by reference.

BACKGROUND 1. Field of the Invention

Example embodiments of the present invention relate to an apparatus andmethod for searching for a building on the basis of an image and amethod of constructing a building search database (DB) for image-basedbuilding search, and more particularly, to an apparatus and method forreceiving a query image, extracting visual features of a buildingincluded in the query image, searching for the building by comparing theextracted features with features of buildings stored in a search DB,which has been constructed through image refinement and the like, andproviding information on the building.

2. Description of Related Art

Image-based building search services provide information on a buildingwhose image is captured by a user. The image-based building searchservice is variously used in location-based services, tourism, theaugmented reality (AR) industry, and the like.

Generally, an image DB of a search target region is constructed fromstreet-view images due to easiness of image acquisition, and animage-based building search service is provided using the constructedimage DB.

However, in many street-view images, several buildings are captured inone scene. Therefore, when a search is performed through image matching,ambiguity arises in determining which building was matched with a queryin a retrieved image when multiple buildings are present. Also, whenthere are too many buildings in one image or many objects other thanbuildings are present, an error occurs in an image search.

To solve these problems, a method of constructing a search DB with onlyimages of single buildings may be taken into consideration. However, themethod requires excessive costs to acquire images and thus isinappropriate.

SUMMARY

Accordingly, example embodiments of the present invention are providedto substantially obviate one or more problems due to limitations anddisadvantages of the related art.

Example embodiments of the present invention provide a method ofsearching for a building on the basis of an image.

The method of searching for a building based on an image may compriseconstructing a building search database (DB); receiving a query imagefrom a user terminal; detecting a region to which a building belongs inthe query image; extracting features of the region detected in the queryimage; and searching the building search DB for a building matching theextracted features.

The detecting of the region to which the building belongs may comprisedetecting or segmenting the region to which the building belongs using abuilding detection model, which is generated using acquired buildingimages and building mask images as training images.

The constructing of the building search DB may comprise acquiringbuilding images and building mask images; refining the building imagesby deleting images in which area ratios of buildings are smaller than areference value; detecting keypoints in the refined images; extractingfeatures of the detected keypoints; and storing the extracted featuresin correspondence with the building images and the building mask images.

The refining of the building images may comprise calculating an arearatio of each individual building included in each of the buildingimages using the building mask images; comparing a maximum of calculatedbuilding-specific area ratios with a preset threshold value; anddeleting a building image in which area ratios have been calculated whena comparison result indicates that a maximum is smaller than thethreshold value.

The extracting of the features may comprise selecting keypointsoverlapping the building mask images from among the detected keypoints;and extracting features of the selected keypoints.

The selecting of the keypoints may comprise selecting keypoints whosesurrounding regions overlap the building mask images from among thedetected keypoints.

The extracting of the features may comprise classifying the selectedkeypoints according to individual buildings and extracting featuresaccording to the individual buildings.

The selecting of the keypoints may comprise an operation of selectingbuildings whose area ratios in the building images exceed a presetthreshold value; and an operation of selecting keypoints whosesurrounding regions overlap regions of the selected buildings in thebuilding mask images.

Example embodiments of the present invention also provide a method ofconstructing a building search database (DB) for image-based buildingsearch.

The method of constructing a building search database (DB) forimage-based building search may comprise acquiring building images andbuilding mask images; refining the building images by deleting buildingimages in which an area ratio of buildings is smaller than a referencevalue; detecting keypoints in the refined images;

extracting features of the detected keypoints; and storing the extractedfeatures in correspondence with the building images and the buildingmask images.

The refining of the building images may comprise calculating an arearatio of each individual building included in each of the buildingimages using the building mask images; comparing a maximum of thecalculated area ratios of buildings with a preset threshold value; anddeleting the building image in which area ratios have been calculatedwhen a comparison result indicates that the maximum is smaller than thethreshold value.

The extracting of the features may comprise selecting keypointsoverlapping the building mask images from among the detected keypoints;and extracting features of the selected keypoints.

The selecting of the keypoints may comprise selecting keypoints whosesurrounding regions overlap the building mask images from among thedetected keypoints.

The extracting of the features may comprise classifying the selectedkeypoints according to individual buildings and extracting featuresaccording to the individual buildings.

The selecting of the keypoints may comprise selecting buildings whosearea ratios in the building images exceed a preset threshold value; andselecting keypoints whose surrounding regions overlap regions of theselected buildings in the building mask images.

Example embodiments of the present invention also provide an apparatusfor searching for a building on the basis of an image.

The apparatus for searching for a building based on an image maycomprise at least one processor; and a memory configured to storeinstructions for instructing the at least one processor to perform atleast one operation.

The at least one operation may comprise constructing a building searchdatabase (DB); receiving a query image from a user terminal; detecting aregion to which a building belongs in the query image; extractingfeatures of the region detected in the query image; and searching thebuilding search DB for a building matching the extracted features.

The detecting of the region to which the building belongs may comprisedetecting or segmenting the region to which the building belongs using abuilding detection model, which is generated using acquired buildingimages and building mask images as training images.

The constructing of the building search DB may comprise acquiringbuilding images and building mask images; refining the building imagesby deleting images in which area ratios of buildings are smaller than areference value; detecting keypoints in the refined images; extractingfeatures of the detected keypoints; and storing the extracted featuresin correspondence with the building images and the building mask images.

The refining of the building images may comprise calculating an arearatio of each individual building included in each of the buildingimages using the building mask images; comparing a maximum of thecalculated area ratios of buildings with a preset threshold value; anddeleting the building image in which area ratios have been calculatedwhen a comparison result indicates that the maximum is smaller than thethreshold value.

The extracting of the features may comprise selecting keypointsoverlapping the building mask images from among the detected keypoints;and extracting features of the selected keypoints.

The selecting of the keypoints may comprise selecting keypoints whosesurrounding regions overlap the building mask images from among thedetected keypoints.

Some example embodiments provide a method of searching for a building onthe basis of an image.

Other example embodiments provide a method of constructing a buildingsearch DB for image-based building search.

Other example embodiments provide an apparatus for searching for abuilding on the basis of an image.

BRIEF DESCRIPTION OF DRAWINGS

Example embodiments of the present invention will become more apparentby describing in detail example embodiments of the present inventionwith reference to the accompanying drawings, in which:

FIG. 1 is a block diagram showing functional modules of an apparatus forsearching for a building on the basis of an image according to anexample embodiment of the present invention;

FIG. 2A is a building image generated to construct a building searchdatabase (DB) according to an example embodiment of the presentinvention;

FIG. 2B is a building mask image generated to construct a buildingsearch DB according to an example embodiment of the present invention;

FIG. 3 is a flowchart of a method of constructing a building search DBby refining image data according to an example embodiment of the presentinvention;

FIGS. 4A to 4C are building images which are deleted in image dataaccording to an example embodiment of the present invention;

FIGS. 5A to 5D are example diagrams illustrating a method of selectingkeypoints of a building according to an example embodiment of thepresent invention;

FIG. 6 shows an example of detecting regions in which buildings exist ina building search process according to an example embodiment of thepresent invention;

FIG. 7 is a flowchart of a method of searching for a building on thebasis of an image according to an example embodiment of the presentinvention;

FIG. 8 is a flowchart of a method of constructing a building search DBfor image-based building search according to an example embodiment ofthe present invention; and

FIG. 9 is a block diagram of an apparatus for searching for a buildingon the basis of an image according to an example embodiment of thepresent invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments of the present invention are disclosed herein.However, specific structural and functional details disclosed herein aremerely representative for purposes of describing the example embodimentsof the present invention, however, the example embodiments of thepresent invention may be embodied in many alternate forms and should notbe construed as limited to example embodiments of the present inventionset forth herein.

Accordingly, while the invention is susceptible to various modificationsand alternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that there is no intent to limit theinvention to the particular forms disclosed, but on the contrary, theinvention is to cover all modifications, equivalents, and alternativesfalling within the spirit and scope of the invention. Like numbers referto like elements throughout the description of the figures.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of the present invention. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(i.e., “between” versus “directly between,” “adjacent” versus “directlyadjacent,” etc.). The terminology used herein is for the purpose ofdescribing particular embodiments only and is not intended to belimiting of the invention. As used herein, the singular forms “a,” “an,”and “the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will be further understood thatthe terms “comprises,” “comprising,” “includes,” and/or “including,”when used herein, specify the presence of stated features, integers,steps, operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

It should also be noted that in some alternative implementations, thefunctions/acts noted in the blocks may occur out of the order noted inthe flowcharts. For example, two blocks shown in succession may in factbe executed substantially concurrently or the blocks may sometimes beexecuted in the reverse order, depending upon the functionality/actsinvolved.

In the present invention, colorants may be any substance includingcoloring and having a property of imparting color, may include bothsubstances that have high opacity and are insoluble in water andsubstances that have high transparency and are soluble in water, and maybe referred to as a pigment, a dye, or the like.

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing functional modules of an apparatus forsearching for a building on the basis of an image according to anexample embodiment of the present invention.

Referring to FIG. 1, an apparatus 100 for searching for a building onthe basis of an image may include a search database (DB) constructionmodule 100 a for constructing a DB 116 required for a building search(referred to as a building search DB), and a search module 100 b forsearching for a building using the constructed building search DB 116and providing information related to the building to a user.

Here, the search DB construction module 100 a may include an image DBgenerator 111, a DB refinement and feature extraction section 112, abuilding search DB constructor 113, and/or a building detection modelgenerator 114.

The image DB generator 111 may collect image data using an externalimage DB and the like and generate and/or store building images andbuilding mask images using the collected image data. The external imageDB may be a search site which provides street-view images (e.g., Google,Daum, and Naver).

Specifically, Google Map (http://www.google.com/maps) provides panoramicimages and depth maps of the panoramic images. Such a depth map mayinclude a normal vector of each plane existing in the correspondingpanoramic image, distance information from a camera, and/orpixel-specific indices of planes to which the corresponding pixelsbelong. Therefore, a perspective image may be generated at an angleaccording to certain intervals in yaw and pitch directions using apanoramic image and used as a building image of the present invention,and a mask image of the building may be generated using the generatedperspective image and information obtained from a depth map (normalvectors, distances from a camera, and pixel-specific plane indices).

Also, some map service users may have constructed three-dimensional (3D)model information of search target regions in advance. In this case, itis possible to generate a building image and a building mask image usingan image (or a panoramic image) and the 3D model information.

For a detailed method of acquiring a building image and a building maskimage, it is recommended to refer to Cavallo, Marco. “3D CityReconstruction From Google Street View.” Comput. Graph. J (2015) andTorii, Akihiko, et al. “24/7 Place Recognition by View Synthesis.”Proceedings of the IEEE Conference on Computer Vision and PatternRecognition (2015).

The DB refinement and feature extraction section 112 may delete anunnecessary building image which degrades search performance using thebuilding images and the building mask images previously generated by theimage DB generator 111, and extract feature information of buildingsusing the refined building images and building mask images.

The building search DB constructor 113 may generate the building searchDB 116 which will be used for building search by storing the buildingfeature information acquired through the DB refinement and featureextraction section 112 and the building images and/or the building maskimages in correspondence with each other. Specifically, the buildingsearch DB constructor 113 may generate the building search DB 116 inwhich the building feature information and the building images and/orthe building mask images are reconfigured in the form of data which iseasily searched. For example, the building search DB 116 may includefile names of images from which the building feature information hasbeen extracted, building-specific feature information, buildingcoordinates in images, and building-specific associated information.Here, the building coordinates in images may be coordinates of boundingboxes including the corresponding building regions in the building maskimages. The building-specific associated information may be detailedbuilding information collected regarding the corresponding buildings.For example, when a building is a restaurant, a menu, reviews, adiscount coupon, etc. may be included, and when a building is a store, astore advertisement, sales items, selling prices, etc. may be included.

Also, place or location information at which building images have beencaptured may be acquired and indexed in the building search DB 116 sothat place-based search may be supported. For example, when a userprovides a query image captured at a specific location (or globalpositioning system (GPS) location information), the building search DB116 may be searched for images within a margin of error based on theprovided specific location.

The building detection model generator 114 may generate a buildingdetection model 115 for detecting a building region in the query imageinput from the user who wants a building search. Specifically, thebuilding detection model generator 114 may generate the buildingdetection model 115 for detecting a region in which a building exists ina specific image using the building images and the building mask imagespreviously generated by the image DB generator 111 as training images.

According to a first example of the building detection model 115, abuilding region (or a building position) in the query image may bedetected in the form of coordinates of a bounding box. In this case,bounding boxes of buildings may be calculated using building maskimages, and the first example of the building detection model 115 may begenerated using building images and coordinates of the bounding boxes astraining data. As a training method, a faster region-based convolutionalneural network (R-CNN) (Shaoqing Ren et al., “Faster R-CNN: towardsreal-time object detection with region proposal networks”, Advances inneural information processing systems, 2015) or a single shot multiboxdetector (SSD) (“Wei Liu et al., “SSD: Single Shot MultiBox Detector”,arXiv:1512.02325, 2015) may be used.

According to a second example of the building detection model 115, abuilding region in the image may be segmented into pixels. In this case,the second example of the building detection model 115 may be generatedusing building images previously obtained through the image DB generator111 and mask images corresponding thereto as training data. As atraining method, a mask R-CNN (Kaiming He et al., “Mask R-CNN”, IEEEInt. Conf. on Comp. Vision, 2017) may be used.

Meanwhile, the search module 100 b may receive a query image, a queryword, and/or the like for a search from a user, search the buildingsearch DB 116, which has been constructed by the search DB constructionmodule 100 a, for a building corresponding to the query image and/or thequery word, and provide an image of the searched building andbuilding-related information to the user.

Specifically, a query image input receiver 121 may receive an image of abuilding which will be searched for, a name of the building, a locationof the building, etc. from the user. When the name, location, etc. ofthe building are received, an image of the building or informationrelated to the building may be immediately provided to the user withreference to names, locations, etc. stored in the building search DB116.

Meanwhile, when an image of the building (referred to as a query imagebelow) is input from the user, a building region detector 122 may detecta building region included in the query image using the buildingdetection model 115 generated by the search DB construction module 100a. Specifically, a bounding box of a region in which the building existsmay be generated in the query image using the detection method of thebuilding detection model 115. Also, a mask image of the building may begenerated in the query image using the segmentation method of thebuilding detection model 115.

A query image feature extractor 123 may extract features of the buildingregion detected by the building region detector 122. When a plurality ofbuilding regions are detected by the building region detector 122,features of the plurality of building regions may be extracted.

An image-based searcher 124 may search the building-specific featureinformation stored in the building search DB 116 for a building havingfeature information which is very similar to the feature informationextracted from the building region. It is possible to provide the userwith building-related information indexed in the building search DB 116regarding the searched building.

Also, when features of a plurality of building regions are extracted bythe query image feature extractor 123, the image-based searcher 124 maysearch for feature information which is very similar to featureinformation of each individual building region. Therefore, the user maybe provided with building-related information of all the plurality ofbuildings included in the query image.

Although it has been described that the search DB construction module100 a and the search module 100 b are included in the apparatus 100 forsearching for a building on the basis of an image, the present inventionis not limited thereto. Each of the search DB construction module 100 aand the search module 100 b may be embodied into a hardware deviceincluding a processor and a memory. In this case, the hardware device ofthe search module 100 b may collect and process data by accessing thebuilding search DB 116 which is constructed separately therefrom.

FIG. 2A is a building image generated to construct a building search DBaccording to an example embodiment of the present invention. FIG. 2B isa building mask image generated to construct a building search DBaccording to an example embodiment of the present invention.

To construct an image DB for building search according to an exampleembodiment of the present invention, building images and building maskimages may be generated. Referring to FIG. 2A, it is possible to see abuilding image, which may be a perspective image generated at an angleaccording to certain intervals in yaw and pitch directions using apanoramic image.

Also, referring to FIG. 2B, it is possible to see a building mask imagegenerated using information obtained from the generated perspectiveimage and a depth map.

FIG. 3 is a flowchart of a method of constructing a building search DBby refining image data according to an example embodiment of the presentinvention. FIGS. 4A to 4C are building images which are deleted in imagedata according to an example embodiment of the present invention.

As described above regarding the DB refinement and feature extractionsection 112 of FIG. 1, a method of searching for a building on the basisof an image according to an example embodiment of the present inventionincludes a process of refining some image data of building images andbuilding mask images and extracting feature information on the basis ofthe refined image data.

Referring to FIG. 3, in the process of refining image data andextracting features, an area ratio of each building in the buildingimages may be calculated using the building mask images (S121). The arearatio may be calculated using Equation 1 below.

$\begin{matrix}{{P\left( b_{i} \right)} = \frac{\Sigma_{({0,0})}^{({W,H})}{m\left( {x,y,b_{i}} \right)}}{W \times H}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Referring to Equation 1, an area (e.g., the number of pixels of a maskimage in the corresponding building) of each building b_(i) (i is anindex of each individual building) in a mask image B(x, y) of thecorresponding building may be calculated on the basis of an area of awhole building image (e.g., the total number of pixels) which is aproduct of a horizontal length W and a vertical length H of the buildingimage so that an area ratio P(b_(i)) may be obtained.

An area of a specific building in a mask image may be calculated usingEquation 2 below.

$\begin{matrix}{{m\left( {x,y,b_{i}} \right)} = \begin{Bmatrix}{1,} & {{{if}\mspace{14mu} {B\left( {x,y} \right)}} = b_{i}} \\{0,} & {{otherwise}.}\end{Bmatrix}} & \left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack\end{matrix}$

Referring to Equation 2 above, m(x, y, b_(i)) is a function whichreturns 1 when pixel values of the building mask image B(x, y)corresponding to coordinates (x, y) belong to the corresponding buildingb_(i), and returns 0 otherwise.

Therefore, it is possible to calculate an area ratio of each individualbuilding in the building images based on the building mask images byapplying a function obtained according to Equation 2 to Equation 1.

After area ratios are calculated, it may be determined whether themaximum of the calculated area ratios of buildings exceeds a predefinedthreshold value (S122). In other words, a building having the largestarea ratio may be determined from among a plurality of buildingsincluded in each building image, and it may be determined whether thebuilding occupies an area corresponding to the predefined thresholdratio value or more in the building image.

Therefore, when the maximum of the calculated area ratios of buildingsdoes not exceed the predetermined threshold value, there are too manybuildings in the building image, buildings are too small in the buildingimage, or the sky, roads, etc. unrelated to buildings are included toomuch in the building image. In this case, the building image may bedeleted (S124).

Referring to FIG. 4A, the building image corresponds to a case in whichthe maximum of area ratios of buildings in the image is 0.20, and it isdifficult to specify any one building.

Referring to FIGS. 4B and 4C, the building images correspond to cases inwhich the maximum of area ratios of buildings in the image is 0.044 and0.0985. In these images, unnecessary vehicles and streets are includedtoo much.

Therefore, when the predefined threshold ratio value is set to 0.25, thebuilding images of FIGS. 4A to 4C are deleted to prevent unnecessarydegradation of search performance.

After some building images are deleted according to the threshold ratiovalue, keypoints of remaining building images may be detected (S123). Asa method of detecting keypoints, it is possible to use a difference ofGaussian (DoG) detector (D. Lowe, “Distinctive image features fromscale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp.91-110, 2004.), a fast-Hessian detector (Herbert Bay, “Speeded-Up RobustFeatures (SURF),” Computer Vision and Image Understanding 110 pp.346-359, 2008), a Laplacian of Gaussian (LoG) detector, aHarris/Hessian-affine detector, or the like.

When keypoints of the building images are detected, keypoints existingin building mask regions may be selected from among the detectedkeypoints (S125). In other words, since the keypoints detected in thebuilding images include keypoints of objects other than buildings,keypoints belonging to building mask regions are selected so that onlykeypoints of buildings may be selected.

When keypoints of buildings are selected, features of the selectedkeypoints may be extracted (S126). As a method of extracting features ofkeypoints, it is possible to use scale invariant feature transform(SIFT) “D. Lowe, “Distinctive image features from scale-invariantkeypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.) orSURF (Herbert Bay, “Speeded-Up Robust Features (SURF),” Computer Visionand Image Understanding 110 pp. 346-359, 2008). In other words, featuresare extracted not from all keypoints in building images but from onlykeypoints corresponding to buildings. Therefore, processing speed may beimproved compared to that of a method of extracting all features from abuilding image and then selecting features of buildings.

FIGS. 5A to 5D are example diagrams illustrating a method of selectingkeypoints of a building according to an example embodiment of thepresent invention.

Referring to FIG. 5A, it is possible to see a building image to whichthe method of selecting keypoints will be applied. From the buildingimage of FIG. 5A, keypoints may be extracted as shown in FIG. 5B. InFIG. 5B, it is possible to see that keypoints overlap a building maskimage, and only the keypoints overlapping the building mask image may beconsidered as keypoints of buildings. Therefore, it is possible toselect keypoints existing in the building mask image.

Even when keypoints exist in a building mask image, an area ratio of thecorresponding building to the image may be small. In this case, searchperformance may be low. Therefore, only keypoints corresponding to abuilding whose area ratio is the predefined threshold value or moreaccording to Equations 1 and 2 above may be selected from among thekeypoints existing in the building mask image.

Meanwhile, it is possible to see that a building mask image does notexactly coincide with an actual building as indicated by a referencenumber 55 of FIG. 5B. Therefore, when only keypoints existing in thebuilding mask image are selected, some of keypoints corresponding to thebuildings may be lost. In consideration of such an error of the buildingmask image, according to an example embodiment of the present invention,when a partial region (e.g., an N×N region) in which keypointspreviously extracted from the building image is centered overlaps thebuilding mask image, the keypoints may be selected as a buildingkeypoints.

Also, according to an example embodiment of the present invention, theselected keypoints may be classified according to buildings and stored.Referring to FIGS. 5C and 5D, it is possible to determine whether a30×30 pixel region in which each keypoint extracted from the buildingimage is centered overlaps the building mask image and see keypointsselected as overlapping keypoints. Here, it is possible to see thatkeypoints have been selected according to buildings.

Like this, when keypoints are stored in a building search DB accordingto buildings, a search is performed on the basis of building-specifickeypoints. Therefore, search speed may be reduced, but any one buildingmay be directly obtained as a search result. On the other hand, whenkeypoints existing in building mask images constitute a building searchDB, a building image having keypoints corresponding to a query image issearched for. Therefore, search speed may be increased, but it isadditionally necessary to verify geometric relationships betweenkeypoints and the like so as to determine which building is a buildingsearched for in a searched building image. Consequently, search speedmay be reduced when information related to the building is required.

FIG. 6 shows an example of detecting regions in which buildings exist ina building search process according to an example embodiment of thepresent invention.

In a method of searching for a building on the basis of an imageaccording to an example embodiment of the present invention, a buildingregion may be detected in a query image, and then features may beextracted. Therefore, it is necessary to detect a building region in aquery image, and the building detection model described above in FIG. 1may be used at this time.

Referring to FIG. 6, it is possible to see that bounding boxesindicating regions in which buildings exist are generated in a queryimage using a building detection model which is generated usingcoordinates of bounding boxes extracted from building mask images andbuilding images as training data.

Only building regions may be detected in the query image on the basis ofthe generated bounding boxes, and it is possible to detect keypoints inthe detected building regions.

FIG. 7 is a flowchart of a method of searching for a building on thebasis of an image according to an example embodiment of the presentinvention. FIG. 8 is a flowchart of a method of constructing a buildingsearch DB for image-based building search according to an exampleembodiment of the present invention.

First, referring to FIG. 7, a method of searching for a building on thebasis of an image may include an operation of constructing a buildingsearch DB (S100), an operation of receiving a query image from a userterminal (S110), an operation of detecting a region to which a buildingbelongs in the query image (S120), an operation of extracting featuresof the region detected in the query image (S130), and an operation ofsearching the building search DB for a building matching the extractedfeatures (S140).

In the operation of detecting a region to which a building belongs(S120), the region to which the building belongs may be detected orsegmented using a building detection model which is generated usingacquired building images and building mask images as training images.

The operation of constructing a building search DB (S100) may beperformed according to the flowchart of FIG. 8.

Referring to FIG. 8, the method of constructing a building search DB forimage-based building search may include an operation of acquiringbuilding images and building mask images (S200), an operation ofrefining the building images by deleting building images in which anarea ratio of buildings is smaller than a reference value (S210), anoperation of detecting keypoints in the refined images (S220), anoperation of extracting features of the detected keypoints (S230), andan operation of storing the extracted features in correspondence withthe building images and the building mask images (S240).

The operation of refining the building images (S210) may include anoperation of calculating an area ratio of each individual buildingincluded in each of the building images using the building mask images,an operation of comparing the maximum of calculated area ratios ofbuildings with a preset threshold value, and an operation of deletingthe building image in which area ratios have been calculated when acomparison result indicates that the maximum is smaller than thethreshold value.

The operation of extracting features (S230) may include an operation ofselecting keypoints overlapping the building mask images from among thedetected keypoints and an operation of extracting features of theselected keypoints.

In the operation of selecting keypoints, keypoints whose presetsurrounding regions overlap the building mask images may be selectedfrom among the detected keypoints.

In the operation of extracting features, the selected keypoints may beclassified according to individual buildings, and features may beextracted according to the individual buildings.

The operation of selecting keypoints may include an operation ofselecting buildings whose area ratios in the building images exceed apreset threshold value and an operation of selecting keypoints whosesurrounding regions overlap regions of the selected buildings in thebuilding mask images.

FIG. 9 is a block diagram of an apparatus for searching for a buildingon the basis of an image according to an example embodiment of thepresent invention.

Referring to FIG. 9, an apparatus 100 for searching for a building onthe basis of an image may include at least one processor 110 and amemory 120 for storing instructions which instruct the at least oneprocessor 110 to perform at least one operation.

The apparatus 100 for searching for a building on the basis of an imagemay include a communication module 130 for receiving a query image or aquery message including a query word from a user terminal owned ormanaged by a user and transmitting a search result to the user terminalvia a wired or wireless network.

The apparatus 100 for searching for a building on the basis of an imagemay further include a storage 140 for temporarily or periodicallystoring and managing data of all or a part of a building search DB andstoring intermediate data of a search process.

The at least one operation may include an operation of constructing abuilding search DB, an operation of receiving a query image from a userterminal, an operation of detecting a region to which a building belongsin the query image, an operation of extracting features of the regiondetected in the query image, and an operation of searching the buildingsearch DB for a building matching the extracted features.

In the operation of detecting a region to which a building belongs, theregion to which the building belongs may be detected or segmented usinga building detection model which is generated using acquired buildingimages and building mask images as training images.

The operation of constructing a building search DB may include anoperation of acquiring building images and building mask images, anoperation of refining the building images by deleting building images inwhich an area ratio of buildings is smaller than a reference value, anoperation of detecting keypoints in the refined images, an operation ofextracting features of the detected keypoints, and an operation ofstoring the extracted features in correspondence with the buildingimages and the building mask images.

The operation of refining the building images may include an operationof calculating an area ratio of each individual building included ineach of the building images using the building mask images, an operationof comparing the maximum of calculated area ratios of buildings with apreset threshold value, and an operation of deleting the building imagein which area ratios have been calculated when a comparison resultindicates that the maximum is smaller than the threshold value.

The operation of extracting features may include an operation ofselecting keypoints overlapping the building mask images from among thedetected keypoints and an operation of extracting features of theselected keypoints.

In the operation of selecting keypoints, keypoints whose presetsurrounding regions overlap the building mask images may be selectedfrom among the detected keypoints.

In the operation of extracting features, the selected keypoints may beclassified according to individual buildings, and features may beextracted according to the individual buildings.

The operation of selecting keypoints may include an operation ofselecting buildings whose area ratios in the building images exceed apreset threshold value and an operation of selecting keypoints whosesurrounding regions overlap regions of the selected buildings in thebuilding mask images.

The apparatus 100 for searching for a building on the basis of an imagemay be, for example, a desktop computer, a laptop computer, a smartphone, a tablet personal computer (PC), a mobile phone, a smart watch,smart glasses, an e-book reader, a portable multimedia player (PMP), aportable game machine, a navigation device, a digital camera, a digitalmultimedia broadcasting (DMB) player, a digital audio recorder, adigital audio player, a digital video recorder, a digital video player,a personal digital assistant (PDA), etc. which are capable ofcommunication.

With the above-described apparatus and method for searching for abuilding on the basis of an image according to the example embodimentsof the present invention, it is possible to construct a DB in units ofbuildings from existing street-view images which are easily obtained.Therefore, search performance may be improved.

Also, a DB is refined by deleting images having regions unnecessary tosearch for a building. Therefore, search accuracy may be improved, andsearch speed may be increased.

While the example embodiments of the present invention and theiradvantages have been described in detail, it should be understood thatvarious changes, substitutions and alterations may be made hereinwithout departing from the scope of the invention.

What is claimed is:
 1. A method of searching for a building based on animage, the method comprising: constructing a building search database(DB); receiving a query image from a user terminal; detecting a regionto which a building belongs in the query image; extracting features ofthe region detected in the query image; and searching the buildingsearch DB for a building matching the extracted features.
 2. The methodof claim 1, wherein the detecting of the region to which the buildingbelongs comprises detecting or segmenting the region to which thebuilding belongs using a building detection model, which is generatedusing acquired building images and building mask images as trainingimages.
 3. The method of claim 1, wherein the constructing of thebuilding search DB comprises: acquiring building images and buildingmask images; refining the building images by deleting images in whicharea ratios of buildings are smaller than a reference value; detectingkeypoints in the refined images; extracting features of the detectedkeypoints; and storing the extracted features in correspondence with thebuilding images and the building mask images.
 4. The method of claim 3,wherein the refining of the building images comprises: calculating anarea ratio of each individual building included in each of the buildingimages using the building mask images; comparing a maximum of calculatedbuilding-specific area ratios with a preset threshold value; anddeleting a building image in which area ratios have been calculated whena comparison result indicates that a maximum is smaller than thethreshold value.
 5. The method of claim 3, wherein the extracting of thefeatures comprises: selecting keypoints overlapping the building maskimages from among the detected keypoints; and extracting features of theselected keypoints.
 6. The method of claim 5, wherein the selecting ofthe keypoints comprises selecting keypoints whose preset surroundingregions overlap the building mask images from among the detectedkeypoints.
 7. The method of claim 5, wherein the extracting of thefeatures comprises classifying the selected keypoints according toindividual buildings and extracting features according to the individualbuildings.
 8. The method of claim 5, wherein the selecting of thekeypoints comprises: an operation of selecting buildings whose arearatios in the building images exceed a preset threshold value; and anoperation of selecting keypoints whose surrounding regions overlapregions of the selected buildings in the building mask images.
 9. Amethod of constructing a building search database (DB) for image-basedbuilding search, the method comprising: acquiring building images andbuilding mask images; refining the building images by deleting buildingimages in which an area ratio of buildings is smaller than a referencevalue; detecting keypoints in the refined images; extracting features ofthe detected keypoints; and storing the extracted features incorrespondence with the building images and the building mask images.10. The method of claim 9, wherein the refining of the building imagescomprises: calculating an area ratio of each individual buildingincluded in each of the building images using the building mask images;comparing a maximum of the calculated area ratios of buildings with apreset threshold value; and deleting the building image in which arearatios have been calculated when a comparison result indicates that themaximum is smaller than the threshold value.
 11. The method of claim 9,wherein the extracting of the features comprises: selecting keypointsoverlapping the building mask images from among the detected keypoints;and extracting features of the selected keypoints.
 12. The method ofclaim 11, wherein the selecting of the keypoints comprises selectingkeypoints whose preset surrounding regions overlap the building maskimages from among the detected keypoints.
 13. The method of claim 11,wherein the extracting of the features comprises classifying theselected keypoints according to individual buildings and extractingfeatures according to the individual buildings.
 14. The method of claim11, wherein the selecting of the keypoints comprises: selectingbuildings whose area ratios in the building images exceed a presetthreshold value; and selecting keypoints whose surrounding regionsoverlap regions of the selected buildings in the building mask images.15. An apparatus for searching for a building based on an image, theapparatus comprising: at least one processor; and a memory configured tostore instructions for instructing the at least one processor to performat least one operation, wherein the at least one operation comprises:constructing a building search database (DB); receiving a query imagefrom a user terminal; detecting a region to which a building belongs inthe query image; extracting features of the region detected in the queryimage; and searching the building search DB for a building matching theextracted features.
 16. The apparatus of claim 15, wherein the detectingof the region to which the building belongs comprises detecting orsegmenting the region to which the building belongs using a buildingdetection model, which is generated using acquired building images andbuilding mask images as training images.
 17. The apparatus of claim 15,wherein the constructing of the building search DB comprises: acquiringbuilding images and building mask images; refining the building imagesby deleting images in which area ratios of buildings are smaller than areference value; detecting keypoints in the refined images; extractingfeatures of the detected keypoints; and storing the extracted featuresin correspondence with the building images and the building mask images.18. The apparatus of claim 17, wherein the refining of the buildingimages comprises: calculating an area ratio of each individual buildingincluded in each of the building images using the building mask images;comparing a maximum of the calculated area ratios of buildings with apreset threshold value; and deleting the building image in which arearatios have been calculated when a comparison result indicates that themaximum is smaller than the threshold value.
 19. The apparatus of claim17, wherein the extracting of the features comprises: selectingkeypoints overlapping the building mask images from among the detectedkeypoints; and extracting features of the selected keypoints.
 20. Theapparatus of claim 19, wherein the selecting of the keypoints comprisesselecting keypoints whose preset surrounding regions overlap thebuilding mask images from among the detected keypoints.